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Executive  Summary 


Interviewers  observed  that  many  respondents  did  not  follow  skip  patterns  on  the 
questionnaire  and  tended  to  ignore  questions  that  appeared  not  to  apply  to  them. 
Respondent  frustration  was  exacerbated  by  the  perceived  repetitiveness  of  questions 
in  certain  areas  and  this  "repetitive  question  fatigue"  is  reflected  in  patterns  of  non- 
response  rates  for  the  NCT. 

Interviewers  also  feel  that  respondents  were  less  likely  to  answer  questions  if  they  are 
not  aware  of,  or  misconstrue,  the  data  requirement.  This  presents  a  dilemma  since 
many  respondents  did  not  consult  the  Guide.  In  fact,  some  interviewers  feel  that 
many  respondents  will  only  do  the  absolute  minimum  required  in  terms  of  reading 
questions  and  instructions. 

The  NCT  questionnaire  was  too  long  and  difficult,  according  to  many  interviewers. 
They  feel  many  respondents  simply  do  not  understand  the  concepts  and  cannot  relate 
to  the  examples  provided.  This  feeling  was  more  prevalent  among  experienced 
interviewers.  A  complete  redesign  to  achieve  a  shorter,  simpler  and  less  demanding 
questionnaire   should  be  contemplated   for  Census  2001. 

According  to  Survey  Operations,  the  NCT  mail  response  rate  was  55.1%.  Without  a 
respondent   information  campaign,  the  NCT  produced  lower  counts  than  the  1991 
Census  for  difficult  to  enumerate    groups,  such  as  new  Canadians.  Based  on  the 
interviewers'  Control  Sheet  information,  the  mail  response  rate  was  15  percentage 
points  higher  for  the  LFS  sample  than  for  the  Special  Population  sample. 

It  is  important  to  note  that  contact  with  the  respondent  at  drop-off  increased  the 
likelihood  of  a  mail-back  response.  If  contact  was  made  during  drop-off,  the  mail 
response  rate  improved  by  over  12%,  according  to  the  Control  Sheet  information.  In 
preparation  a  for  Centralized  Edit  collection  regime,  strategies  to  improve  the  mail 
response  rate  will  have  to  be  examined,  especially  in  the  area  of  communications. 
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1.  Introduction 

In  preparing  for  the  1996  Census  of  Population,  Statistics  Canada  conducted  a  test  of  the 
proposed  questionnaire   content  and  of  selected  field  procedures  and  support  activities  in 
November  of  1993.  As  part  of  this  National  Census  Test  (NCT),  a  debriefing  program  was 
designed  to  solicit  from  the  field  staff  an  appraisal  of  the  questionnaire  and  field  procedures. 
Interviewers  are  on  the  "front  lines"  of  data  collection,  a  unique  position  to  provide  feedback 
on  the  questionnaire  and  testing  procedures.  The  interviewer  debriefing  program  consisted 
of  two  integral  components.  The  first  consisted  of  discussion  sessions  attended  by  field  staff 
and  Head  Office  representatives.  NCT  Report  5  (Interviewer  Debriefing  Report)  documents 
the  germane  points  and  issues  raised  during  these  discussions. 

The  second  component  of  the  debriefing  program  was  an  Interviewer  Debriefing 
Questionnaire  completed  by  Interviewers  and  Senior  Interviewers  upon  the  conclusion  of 
their  assignments  (see  Appendix  A).  This  questionnaire,  however,  was  distributed  prior  to 
data  collection  to  encourage  meaningful  participation  through  an  awareness  of  the  debriefing 
program.  The  first  section  of  this  report  provides  an  overview  of  these  questionnaires.  While 
the  NCT  can  assess  selected  field  procedures  and  support  activities,  it  is  virtually  impossible 
to  replicate  the  Census  ambience  as  part  of  the  testing  environment.  The  second  part  of  this 
report  examines  the  mail-back  response  rates  for  the  NCT  using  the  interviewer  Control 
Sheets  captured  and  appended  to  the  household  data  files. 


2.  Interviewer  Debriefing  Questionnaires 

A  total  of  263  debriefing  questionnaires  were  captured  in  Head  Office  and  the  frequencies 
for  each  question  are  contained  in  Appendix  B.  There  were  only  two  fields  from  the 
questionnaire  which  were  able  to  generate  breakdown  variables  (Table  1).  The  Regional 
Office  and  Sample  Component  variables  are  derived  from  the  Assignment  Number  field  and 
the  Interviewer  Experience  variable  is  generated  directly  from  the  "Completed  by"  question. 
However,  it  is  not  possible  to  differentiate  between  Interviewers  and  Senior  Interviewers 
from  the  Assignment  Numbers. 

There  was  some  variation  in  Interviewer  Experience  by  Regional  Office  (RO).  The 
percentage  of  first-time  interviewers  was  11.1%  in  Montreal  and  14.3%  in  St.  John's  while 
it  was  highest  at  38.1%  in  Vancouver.  The  other  ROs  hovered  around  the  average  of  24% 
first-time  interviewers.  Each  of  the  questions  was  broken  down  by  the  three  variables  in 
Table  1.  The  discussions  which  follow  will  only  refer  to  variation  by  these  three  variables  if 
the  differences  were  found  to  be  both  substantive  and  statistically  significant  at  the  95%  level 
of  confidence. 


Table  1 
Breakdown  Variables 


[1] 


[2] 


N  =  263  [1] 

f 

(%) 

Regional  Office 

St.  John's 

9 

3.4 

Halifax 

63 

24.0 

Montreal 

55 

20.9 

Sturgeon  Falls 

19 

7.2 

Toronto 

32 

12.2 

Edmonton 

41 

15.6 

Vancouver 

23 

8.7 

(unknown)  [2] 

21 

8.0 

Sample  Component 

LFS  Rotations  [2] 

238 

90.5 

Special  Population 

25 

9.5 

Interviewer  Experience 

First-Time 

63 

24.0 

Experienced 

178 

67.7 

(not  reported) 

22 

8.4 

Of  292  interviewers  and  29  senior  interviewers,  263  Interviewer  Debriefing 
Questionnaires  were  completed  and  captured.  It  is  important  to  note  that 
some  interviewers  had  more  than  one  assignment.  There  are  308  unique 
assignment  numbers  on  the  household  data  files. 

There  were  21  cases  in  which  the  Assignment  Number  was  invalid  and  these 
assignments  are  included  with  LFS  rotations  for  the  Sample  Component. 

It  is  important  to  recognize  in  interpreting  the  debriefing  questionnaires  that  interviewers 
formed  opinions  regarding  most  issues  based  on  either  personal  or  telephone  follow-up  with 
respondents.  These  respondents  are  likely  to  be  more  representative  of  uncooperative 
sample  households.  Moreover,  in  many  cases  interviewers  would  complete  the  NCT 
questionnaire  over  the  telephone.  The  NCT  questionnaire  was  not  designed  to  be  an 
interviewer-administered   survey  but  rather  a  respondent-completed    survey.  This  report  will 


cite  two  non-response  rates  from  NCT  Report  10  (Edit  Failure  and  Response  Rates).  The 
"before"  follow-up  rate  is  based  on  a  sample  of  questionnaires  captured  before  field  edit  for 
the  Edit  Failure  Study  (EFS).  The  "after"  follow-up  rate  will  refer  to  the  Labour  Force 
Survey  (LFS)  weighted  non-response.  In  both  cases,  the  "definite"  non-response  rate  is  used. 

Questionnaire   Design 

According  to  most  interviewers,  the  "Step"  process  seemed  to  work  well  and  the 
questionnaire  was  easy  for  respondents  to  follow.  However,  close  to  50%  of  the  interviewers 
responded  that  the  skip  patterns  were  not  clear.  This  problem  is  well  documented  by  other 
NCT  Reports  including  Report  5  (Interviewer  Debriefing  Report)  and  Report  9  (Evaluation 
of  Design  and  Layout). 

About  one  half  of  the  interviewers  responded  that  they  had  rephrased  the  questions  to  make 
them  simpler  or  discussed  the  question  meanings  with  respondents.  Experienced  interviewers 
were  more  likely  to  have  done  so  than  were  their  first-time  counterparts.  It  should  be  noted 
that,  during  training,  interviewers  were  instructed  to  read  the  question  as  it  was  written  on 
the  NCT  questionnaire.  When  completing  questionnaires  for  respondents,  the  likelihood  of 
interviewers  asking  all  the  questions  down  the  page  for  one  person  at  a  time  or  asking  the 
questions  across  the  page  for  all  persons  at  the  same  time  was  more  or  less  the  same.  In  at 
least  one  instance,  an  interviewer  completed  Questions  1  to  28  across  the  page  for  all 
persons  and  then  asked  Questions  28  to  46  down  the  page  for  one  person  at  a  time. 

Front  Page 

Nearly  30%  of  interviewers  reported  getting  a  reaction  to  the  front  cover  message  on  the 
NCT.  Most  of  these  reactions  concerned  the  mandatory  nature  of  the  NCT  and  the  penalty 
for  non-compliance.  There  was  also  much  distrust  regarding  the  confidentiality  of  the 
questionnaire.  Almost  one  in  five  interviewers  provided  a  suggestion  to  improve  the  front 
cover.  Many  suggestions  mentioned  the  need  for  space  on  the  questionnaire  to  record 
follow-up  procedures.  It  appeared  that  many  respondents  misinterpreted  "November  8"  as 
a  deadline  and  often  discarded  the  questionnaire  after  the  date  had  passed. 

Coverage-Related   Questions  (Steps  2  to  6) 

Over  20%  of  interviewers  reported  respondent  difficulties  regarding  whether  or  not  a  person 
should  be  included  as  a  usual  resident.  The  most  frequently  cited  problems  involved  the 
decision  of  whether  to  include  students  who  are  away  at  school,  lodgers  and  relatives  staying 
temporarily.  NCT  Report  11  (Coverage-Related  Steps)  examines  in  more  detail  the 
problems  encountered  with  these  steps. 

Family,  Demography  and  Activity  Limitations  (Questions  2  to  8) 

About  42%  of  interviewers  responded   that  questions  in  this  section  of  the  NCT  caused 


problems  or  were  difficult  for  respondents  to  answer.  Many  interviewers  mentioned 
respondent  frustration  with  having  to  complete  Questions  5  (Marital  Status)  and  6  (Common 
Law)  for  children  under  15  years  of  age.  For  Question  7  (Activity  Limitations)  many 
interviewers  wondered  why  the  same  question  has  to  be  asked  three  times  ^According  to 
NCT  Report  10,  the  before  follow-up  (EFS)  non-response  rate  mcreased  from  5.4%  for 
Question  7a  (Activity  Limitation  -  Home)  to  11.4%  for  Question  7c  (Activity  Limitation  - 
Other)  This  is  the  first  example  of  respondent  "repetitive  question  fatigue  on  the 
questionnaire.  One  in  five  interviewers  reported  that  terms  in  this  section  of  the  NCT  were 
not  well  understood  by  respondents,  most  citing  the  definition  of  "limited  in  Question  7  as 
the  problem.  Compared  with  inexperienced  staff,  experienced  interviewers  were  more  likely 
to  respond  that  such  terms  were  not  well  understood. 

Language  (Questions  9  to  11) 

An  overwhelming  majority  of  interviewers  reported  no  concerns  regarding  the  language 
questions  on  the  NCT.  Perhaps  the  only  noteworthy  comment  is  that  of  respondent 
uncertainty  regarding  the  applicability  of  these  questions  to  infants. 

Socio-Cultural  Information  (Questions  12  to  20) 

This  section  of  the  NCT  resulted  in  both  problems  and  negative  reactions,  according  to  the 
interviewers  Over  40%  of  interviewers  reported  questions  that  caused  problems  or 
difficulties  while  33%  indicated  they  encountered  negative  reactions.  A  common  concern 
cited  by  interviewers  was  the  apparent  repetition  of  NCT  Questions  17  (Aboriginal)  19 
(Band/First  Nation)  and  20  (Registered  Indian).  Many  respondents  inquired  as  to  why  three 
questions  were  necessary  to  identify  Aboriginal  persons.  One  interviewer  noted  that  in 
Question  19,  the  physical  separation  between  the  "Yes"mark-in  and  the  write-in  box  was  too 
Seat  resulting  in  many  missing  write-ins.  According  to  NCT  Report  17  (Aboriginal),  this  is 
of  concern  since  no  other  information  on  the  questionnaire  can  be  used  for  imputation. 

Respondents   were  uncertain  of  "ethnic  or  cultural  group"  in  Question  16  (Ancestry)  either 
in  terms  of  their  own  ancestry  or  the  number  of  generations  to  consider.  One  interviewer 
mentioned  the  uncertainty  if  respondents  or  their  family  members  were  adopted.  Frustration 
also  resulted  from  questions  in  this  section  which  often  did  not  apply  to  respondents.  For 
example,  many  interviewers  wondered  why  respondents  who  answered  "Canadian  by  Brrth 
to  Question  13  (Citizenship),  then  had  to  answer  Question  14  (Landed  Immigrant).  NCT 
Report   15  (Place  of  Birth,  Citizenship  and  Immigration)   noted  the  high  level  of  non- 
response  to  Question    14  among  Canadian   Citizens.  Interviewers  also  wondered   why  a 
respondent   was  required  to  answer  NCT  Questions   19  and  20  after  answering    No  to 
Question  17  and  "White"  to  Question  18  (Population  Group). 

Several  interviewers  commented  that  many  respondents  simply  skipped  Questions  14, 19  and 
20  since  they  did  not  seem  to  apply  to  them  and  this  observation  is  reflected  m  the  before 
follow-up  (EFS)  non-response  rates  of  6.4%,6;2%  and  7.0%  respectively  (NCT  Report  10). 
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This  concurs  with  the  finding  of  NCT  Report  5  that  respondents  tend  not  to  answer  a 
question  which  does  not  concern  them,  leading  to  follow-up.  While  most  interviewers  did  not 
report  problems  with  the  skip  patterns  in  this  section,  this  is  at  odds  with  a  finding  of  NCT 
Report  17.  According  to  the  Report,  55%  of  respondents  indicating  "Yes"  to  Question  17 
failed  to  follow  the  skip  and  continued  with  Question  18. 

In  terms  of  negative  reactions  from  respondents,  interviewers  reported  that  most  were 
directed  at  Question  16  (Ancestry).  Many  respondents  wanted  to  know  why  this  information 
is  collected.  This  observation  confirms  the  finding  of  NCT  Report  25  (Evaluation  of 
Respondents'  Comments)  that  respondents  find  Question  16  difficult  and  objectionable. 
Significantly  fewer  negative  reactions  were  directed  at  Question  18,  perhaps  because  of  the 
accompanying  explanatory  note.  According  to  NCT  Report  10,  the  before  follow-up  non- 
response  rate  for  Question  16  was  12.3%  and  for  Question  18  was  1.7%.  These  rates  were 
reduced  to  3.8%  and  0.3%  respectively  after  follow-up. 

There  was  significant  variation  in  reporting  negative  reactions  by  Interviewer  Experience  and 
Regional  Office.  Experienced  interviewers  were  more  likely  to  report  difficulties  with,  and 
negative  reactions  to,  questions  in  the  Socio-cultural  information  section.  In  terms  of 
interviewers  reporting  problems  and  difficulties,  responses  ranged  from  a  low  of  22%  m 
Sturgeon  Falls  to  a  high  of  75%  in  St.  John's.  As  for  negative  reaction,  these  questions  were 
less  likely  to  encounter  negative  reactions  in  the  Montreal,  Toronto  and  Vancouver  ROs. 
Perhaps  this  is  to  be  expected  since  these  larger  urban  areas  are  likely  to  expose  persons  to 
more  ethnic  diversity,  hence  the  question  appears  to  be  more  applicable  to  the  respondent 
population. 

As  a  final  note,  the  NCT  found  the  proportion  of  immigrants  lower  than  expected  compared 
with  the    1991    Census    (NCT   Report    15).   This   underscores    the    importance     of  a 
communications  program  to  heighten  public  awareness.  A  respondent  information  campaign 
targetted  at  difficult  to  enumerate  groups  such  as  immigrants  and  non-permanent   residents 
is  of  particular  importance. 

Mobility  (Questions  21  and  22) 

There  was  nothing  that  appeared  to  be  controversial  or  difficult  about  this  section  of  the 
NCT  questionnaire,  according  to  the  interviewers. 

Education  (Questions  23  to  25) 

Approximately  14%  of  interviewers  reported  problems  with  these  questions.  The  vast 
majority  of  these  problems  centre  around  two  issues.  First,  many  respondents  reported  then- 
highest  level  of  education  obtained  rather  than  all  of  their  qualifications.  Second,  immigrants 
found  the  listed  qualifications  incongruent  with  what  they  obtained  in  other  countries.  As 
such,  interviewers  reported  that  immigrants  often  indicated  "None"  for  this  question.  This 
issue  requires  further  attention. 
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Household  and  Volunteer  Activities  (Question  26  and  27) 

Over  30%  of  interviewers  reported  that  questions  in  this  section  caused  both  problems  and 
negative  reactions.  Although  the  examples  provided  on  the  NCT  questionnaire  were  found 
to  be  quite  helpful,  interviewers  reported  that  since  most  people  do  not  keep  track  of  time 
spent  on  these  activities,  it  was  extremely  difficult  and  time-consuming  to  estimate  the  tune 
spent  over  the  past  seven  days  to  answer  Question  26  (Household  Activities).  Dividing 
Question  26  into  four  parts  simply  exacerbated  the  frustration  and,  as  a  result,  many 
respondents  simply  left  26d  blank.  According  to  NCT  Report  10,  the  before  follow-up  (EFS) 
non-response  rate  increased  from  5.4%  for  Question  26a  (Unpaid  -Housework)  to  9.9%  for 
Question  26d  (Unpaid  -  Care/Others).  This  may  be  another  example  of  "repetitive  question 
fatigue"  on  the  NCT  questionnaire  or  a  reflection  of  inappropriate   response  categories. 

Interviewers  noted  that  the  answers  provided  by  respondents  were  inconsistently  reported. 
For  example,  while  one  respondent  would  carefully  calculate  the  hours  spent  caring  for  their 
children  another  respondent  would  maintain  that  caring  for  children  is  a  24  hours  per  day 
endeavour    Moreover,  several  interviewers  noted  this  inconsistency  when  a  respondent 
provided   answers   for  other   members    of  Ihe  household    (i.e.  proxy  responses).    Such 
inconsistency  calls  into  question  the  validity  of  these  data  in  the  statistical  sense. 

Interviewers  reported  that  many  respondents  found  these  questions  to  be  intrusive  and 
questioned  the  need  for  collecting  such  data.  Again,  this  underlies  the  notion  that 
cooperation  is  related  to  an  understanding  of  the  data  requirement.  One  interviewer  noted 
that  Seniors  strongly  objected  to  Question  26c. 

While  just  over  10%  of  interviewers  responded  that  terms  in  this  section  were  not  well 
understood  by  respondents,  experienced  interviewers  were  more  likely  than  first-time 
interviewers  to  report  this  observation.  Such  a  difference  was  also  observed  for  terms  used 
in  the  Family  Demography  and  Activity  Limitations  section.  However,  it  is  important  to  note 
that  experienced  interviewers  were  more  likely  to  have  rephrased  the  NCT  questions  and 
to  have  discussed  the  meaning  of  questions  with  respondents.  Such  interviewer  intervention 
makes  it  virtually  impossible  to  distinguish  genuine  respondent  difficulties  with  designed 
terminology  from  the  effect  of  experienced  interviewers'  perception  of  and  rephrasing  of 
terms  and  concepts.  In  these  cases,  focus  group  testing  may  well  be  a  better  arbitrator  of 
respondent  understanding  of  terms  and  concepts. 

Labour  Market  Activities  (Questions  28  to  45) 

A  majority  (57.2%)  of  interviewers  reported  that  questions  in  this  section  of  the  NCT  caused 
problems  or  were  difficult  for  respondents  to  answer.  It  is  important  to  note  that  this  is  not 
a  question-specific  difficulty. The  interviewers  reported  that  respondents  find  that  this  section 
contains  a  high  number  of  repetitive  questions.  For  example,  many  interviewers  noted  the 
frustration  of  respondents  with  Questions  34,  35,  36, 37  and  38.  According  to  NCT  Report 
10,  the  before  (EFS)  and  after  (LFS  weighted)  follow-up  non-response    rates  for  these 
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questions  were  as  follows: 

Question  34  (Name  of  Employer)  9.6%  -»  2.9% 

Question  35  (Kind  of  Business)  13.0%  -»  4.0% 

Question  36  (Industry)  16.0%  -»  4.2% 

Question  37  (Kind  of  Work)  12.1%  ->  3.0% 

Question  38  (Important  Duties)  19.0%  -»  6.2%  . 

This  is  another  example  of  "repetitive  question  fatigue"  in  that  respondents  are  less  receptive 
to  the  second  and  subsequent  questions  aimed  at  the  same  perceived  concept  (industry  and 
occupation  in  this  case).  While  it  is  difficult  to  precisely  quantify  the  impact  of  such  apparent 
respondent  fatigue  on  follow-up,  the  before  (EFS)  follow-up  non-response  rates  for  these 
questions  indicate  that  they  contributed  significantly  to  edit  failure. 

Many  interviewers  commented  on  the  level  of  language  used  in  this  section  and  wondered 
how  Statistics  Canada  could  expect  people  to  understand  the  questions  and  relate  to  the 
categories  and  examples  provided.  For  example,  the  second  write-in  block  for  Question  34 
was  more  often  than  not  left  blank  by  respondents.  NCT  Report  8  (Rapport  sur  le  niveau 
de  langue  des  questionnaires)  examines  the  level  of  language  concern  and  finds  that  labour 
market  questions,  while  difficult,  are  less  so  than  the  education  and  income  questions  in  both 
official  languages.  It  is  important  to  note  that  NCT  Report  8  measures  the  level  of  language 
as  a  function  of  the  grammatical  structure  of  the  text  irrespective  of  the  sequence  of 
questions  length  or  the  skip  patterns  contained  therein. 

The  Labour  Market  Activities  section  appeared  to  result  in  some  confusion  and  frustration 
for  respondents.  Several  interviewers  noted  that  many  respondents  simply  stopped 
completing  the  questionnaire  in  this  section,  resulting  in  the  need  for  interviewer  follow-up; 
a  point  raised  in  NCT  Report  5.  While  over  30%  of  interviewers  encountered  negative 
reactions  from  respondents  to  questions  in  this  section,  there  is  no  single  question  or  series 
of  questions  which  attracted  this  reaction.  It  is  important  to  note  that,  compared  with  the 
LFS  sample,  interviewers  working  with  Special  Population  assignments  were  more  likely  to 
report  problems/difficulties  and  negative  reactions. 

Most  interviewers  (53.6%)  reported  that  respondents  failed  to  correctly  follow  the  skip 
patterns  in  the  Labour  Market  section.  Many  respondents  missed  the  skip  in  Question  28 
(Hours  Worked  for  Pay)  while  others  entered  "0" hours  in  the  write-in  box  and  erroneously 
followed  the  skip.  Several  interviewers  suggested  that  reversing  the  write-in  box  and  the 
"None"  mark-in  circle  in  Question  28  would  improve  the  situation.  Other  interviewers 
wondered  why  there  is  no  mark-in  circle  to  steer  retired  seniors  around  some  of  these 
questions  since  seniors  found  answering  Questions  28  to  33  frustrating.  Moreover,  given  the 
problem  respondents  had  following  skip  patterns,  many  respondents,  including  seniors,  would 
have  inadvertently  missed  the  skip  in  Question  33. 

NCT  Report  21  (Labour  Market  Activities)  examined  the  comments  provided  by  households 
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in  Step  10  and  concludes  that  the  three  main  areas  of  concern  are  the  frustration  of  retired 
persons  with  Questions   38-45,  the  confusion  over  skip  instructions  and  the  difficulty  for 
multiple  job  holders.  These  problems  are  likely  to  become  more  prevalent  as  the  population 
ages  and  as  non-traditional  patterns  of  education  and  labour  force  participation  grow  (e.g. 
part-time   multiple  jobs,  semi-retirement).  Although  this  section  appears  to  have  resulted  m 
difficulties  for  respondents  according  to  the  interviewers,  NCT  Report  25,  which  examined 
respondents'  comments  on  the  NCT,  seems  not  to  have  registered  a  similar  level  of  concern 
for  several  reasons.  To  begin,  there  is  no  single  "target"  question  in  the  Labour  Market 
Activities  section  which  is  identified  by  respondents  as  being  difficult  and/or  objectionable. 

Moreover,  the  Labour  Market  Activities  section  is  followed  by  Question  46  (Income)  which 
tends  to  attract  a  higher  level  of  negative  reaction  and  occurs  just  prior  to  respondents 
completing  Step  10  of  the  NCT  questionnaire.  Finally,  interviewers  noted  that  many 
respondents  failed  to  register  their  concerns  by  not  completing  Step  10.  However,  these 
observations  do  not  lessen  the  problems  cited  in  this  report.  It  would  appear  that  a  complete 
redesign  of  the  Labour  Market  Activities  section  should  be  contemplated  for  the  2001 
Census  to  reduce  respondent  burden.  If  successful,  this  redesign  could  reduce  the  incidence 
of  follow-up  and  processing  contingencies  which  should,  concomitantly,  improve  data  quality. 
Such  a  redesign  should  include  a  filter  question  to  identify  individual  situations  so  that 
certain  respondent  groups  (i.e.  retired  seniors)  would  be  directed  to  specific  questions. 

Income  (Question  46) 

A  clear  majority  (63.6%)  of  interviewers  indicated  that  respondents  had  difficulties  with  the 
income  question.  This  ranged  from  a  low  of  44%  in  the  Sturgeon  Falls  RO  to  a  high  of  78% 
in  the  Vancouver  RO.  A  vast  majority  (95%)  of  interviewers  reported  encountering  negative 
reactions  to  this  question  from  respondents.  Experienced  interviewers  were  more  likely  than 
were  first-time  interviewers  to  report  such  a  respondent  reaction.  The  difficulties  with,  and 
objections  to,  this  question  are  weU  documented  (see  NCT  Report  25  for  example).  They 
are  also  reflected  in  the  NCT  non-response  rates.  The  before  (EFS)  follow-up  non-response 
rate  for  income  components  ranged  from  a  low  of  23.6%  for  Question  46a  (Wages  & 
Salaries)  to  a  high  of  32.5%  for  Question  46j  (Other  Income).  These  rates  were  reduced  by 
follow-up  to  11.3%  and  18.4%  respectively  (LFS  weighted). 

Dwelling  (Question  47  and  48) 

According  to  the  interviewers,  this  section  of  the  NCT  questionnaire   did  not  generate  a 
significant  number  of  difficulties  or  negative  reactions. 

Field  Operations 

The  42  5%  of  interviewers  reporting  that  respondents  wanted  to  complete  the  questionnaire 
during  drop-off  indicated  that  an  average  of  just  over  5  respondents  per  assignment  did  so. 
Compared  with  new  staff,  experienced  interviewers  were  more  likely  to  have  made  this 
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observation.  Again,  the  cohort  of  experienced  interviewers  appears  to  be  perceiving 
respondent  behaviour  differently  than  did  their  first-time  counterparts.  An  overwhelming 
majority  of  interviewers  found  the  pictures  and  definitions  of  dwelling  codes  to  be  helpful 
and  easily  understood.  Interviewers  in  the  Montreal,  Toronto  and  Vancouver  ROs  were  less 
likely  to  use  the  Dwelling  Type  Chart  on  the  reference  card. 

Most  interviewers  referred  to  the  Interviewer's  Manual  during  enumeration  and  found  it  easy 
to  understand  and  the  answers  relatively  quick  to  find.  A  majority  also  did  not  find  the 
procedural  definitions  confusing  and  found  the  edit  steps  easy  to  understand  and  the  rules 
for  determining  follow-up  requirements  clear.  However,  close  to  40%  of  interviewers  made 
suggestions  that  they  felt  would  make  the  manual  a  better  reference  tool  and  easier  to  use. 
The  most  frequent  suggestion  was  the  use  of  some  kind  of  index  or  colour-coded  tabs  to 
improve  accessing  the  manual's  contents.  While  several  interviewers  suggested  changing  the 
sequence  to  improve  the  manual,  one  interviewer  stated  that  familiarity,  regardless  of  the 
sequencing,  is  a  prerequisite  for  the  manual's  effectiveness. 

A  slight  majority  of  interviewers  favoured  completing  the  editing  for  one  person  at  a  time 
rather  than  on  a  question-by-question  basis.  One  half  of  the  interviewers  indicated  that  they 
referred  to  the  edit  steps  at  the  beginning  while  editing  questionnaires.  One  interviewer 
insisted  on  contact  both  at  drop-off  and  pick-up  so  that  editing  could  be  done  with  the 
respondents  present.  There  was  variation  by  Regional  Office  in  the  response  to  these 
questions,  perhaps  reflecting  differences  in  training.  Finally,  42%  of  interviewers  reported 
that  other  tools  would  have  helped  them  do  a  better  job.  The  most  frequently  mentioned 
item  was  to  improve  the  maps  used  in  the  field.  NCT  Report  6  (Evaluation  of  Field 
Collection  Procedures)  examines  many  of  these  issues  in  greater  detail. 

Other  Comments 

Of  the  263  questionnaires  received  by  Head  Office,  202  included  "other  comments". 
However,  many  of  these  comments  were  not  always  directed  at  the  NCT  questionnaire  per 
se.  The  comments  ranged  from  suggestions  for  the  construction  of  specific  assignment  areas 
in  Regional  Offices  to  an  expression  of  appreciation  to  Pamela  White  for  her  visit  with 
interviewers  and  her  attentiveness  to  their  concerns.  Of  the  themes  that  tended  to  recur, 
they  can  be  divided  into  test-specific  and  other  comments. 

Test-specific  comments  tend  to  be  concerns  or  suggestions  arising  from  the  fact  the  NCT  was 
a  survey  and  not  an  actual  census.  For  example,  the  use  of  an  introductory  letter  explaining 
the  NCT  and  its  mandatory  nature  would  have  been  useful  when  contact  was  not  made 
during  drop-off.  This  letter  should  also  mention  that  follow-up  occurs  if  the  questionnaire 
is  not  returned  by  mail.  Many  interviewers  felt  that  an  awareness  of  potential  follow-up 
might  have  encouraged  respondents  to  promptly  complete  the  questionnaire  in  many  cases. 
Another  frequent  comment  concerned  the  problem  of  accessing  apartment  buildings,  while 
several  comments  mentioned  that  the  training  time  was  inadequate,  especially  for  new 
interviewers.  A  number  of  interviewers  from  the  Prairie  region  mentioned  that  working  on 
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Sunday  was  a  problem  with  respondents. 

A  number  of  interviewers  mentioned  that  there  was  inadequate  time  allowed  before 
telephone  follow-up  and  problems  stemmed  from  holding  the  questionnaire  at  the  ROs. 
There  appeared  to  be  numerous  occasions  in  which  an  interviewer  completed  a 
questionnaire  over  the  phone  only  to  have  the  original  questionnaire,  which  the  respondent 
insisted  was  in  the  mail,  arrive  the  following  day.  This  problem  was  more  acute  in  the  Prairie 
Region  where  mail  could  sometimes  take  up  to  7  days  to  travel  from  Manitoba  to  the 
Edmonton  RO.  Interviewers  mentioned  that  this  situation  created  a  negative  public  relations 
image  for  Statistics  Canada. 

It  should  be  noted,  however,  that  these  test-specific  concerns  should  be  alleviated  during  the 
actual  Census  either  because  of  operational  differences  or  heightened  public  awareness. 
Several  interviewers  suggested  that  the  Census  test  should  be  conducted  in  the  summer  to 
take  advantage  of  more  daylight  hours.  As  a  final  note  on  test-specific  comments,  there  were 
relatively  few  complaints  from  respondents  about  having  recently  been  selected  for  the 
Labour  Force  Survey. 

For  comments  that  were  not  of  a  test-specific  nature,  most  related  to  questionnaire  length 
and  difficulty  and  several  interviewers  mentioned  that  some  respondents  were  simply 
intimidated.  Many  interviewers  mentioned  that  the  mail-back  response  was  much  greater  for 
households  in  which  personal  contact  with  the  respondent  was  made  during  drop-off  (see 
Table  3).  One  interviewer  noted  that  she  had  to  complete  the  questionnaire  with  a  senior 
citizen  who  insisted  she  could  not  have  completed  the  survey  without  assistance.  One 
interviewer  felt  that  most  respondents  complied  with  the  NCT  only  because  it  was 
mandatory.  Respondents  would  not  spend  time  to  read  the  Guide  or  use  the  Census  Help 
Line  (CHL).  This  same  interviewer  suggested  that  respondents  even  avoided  expressing  then- 
sentiments  in  writing  (NCT  Step  10).  Another  interviewer  made  the  following  observation: 

"I  also  found  that  Step  10  was  answered  with  very  little  thought  if  not  omitted 
entirely.  Even  when  it  couldn't  have  been  more  obvious  that  the  respondent 
did  not  understand  parts  of  the  questionnaire,  they  would  answer  "No"  to  Step 
10  Questions." 

Perhaps  this  should  be  kept  in  mind  when  interpreting  the  incidence  rates  reported  in  NCT 
Report  4  (Census  Help  Line)  and  NCT  Report  25.  One  interviewer  strongly  recommended 
that  the  CHL  number  be  prominently  displayed  on  the  front  cover. 

Several  interviewers  stressed  the  need  for  Statistics  Canada  to  explain  the  reasons  why  these 
data  are  needed.  While  this  is  the  function  of  the  Census  Guide,  many  respondents  do  not 
read  the  Guide.  For  the  1996  Census,  the  decision  to  put  the  reasons  for  questions  directly 
on  the  short  questionnaire  (form  2A)  and  eliminate  the  Guide  (form  9A)  should  improve 
this  situation.  Finally,  one  interviewer  strongly  suggested  that  Statistics  Canada  use 
meaningful  examples,  to  which  the  average  person  can  relate,  on  how  the  data  they  provide 
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fit  into  the  overall  picture.  This  concurs  with  a  conclusion  of  NCT  Report  25  that  people  will 
be  more  likely  to  provide  the  requested  information  if  they  understand  why  it  is  collected. 

3.  Mail  Response  and  the  Test  Environment 

This  section  of  the  report  examines  the  NCT  mail  response  rates  and  factors  which  affect 
these  response  rates.  One  consideration  motivating  this  line  of  inquiry  is  the  fact  that  the 
mail  response  rate  for  the  decennial  United  States  Census  declined  from  75%  m  1980  to 
65%  in  1990,  resulting  in  significant  expenditure  increases  for  follow-up.  This  is  of  concern 
given  that  the  1996  Census  is  conducting  a  Centralized  Edit  test  in  preparation  for  Census 
2001  and  the  mail  response  rate  is  an  acute  consideration  in  such  an  environment.  According 
to  Survey  Operations  Division  (SOD),  the  final  mail  response  rate  for  the  NCT  was  55.1% 
(9  954  questionnaires  returned  from  a  total  drop-off  of  18,160);  this  excludes  out-of-scope 
dwellings  from  the  original  drop-off  (21,107  dwellings).  It  should  be  noted  that  the  NCT  was 
conducted  without  the  media  awareness  campaign  which  normally  accompanies  a  census. 

Analysis  of  Control  Sheet  Data 

An  interviewer  Control  Sheet  was  provided  for  each  assignment  in  the  NCT  so  that  the 
interviewer  could  record  survey  information  pertaining  to  each  dwelling  (e.g.  number  of 
follow-up  attempts).  These  Control  Sheets  were  data  captured  and  appended  to  the 
household  files.2  However,  an  investigation  of  the  Control  Sheet  data  seems  to  indicate  that 
there  were  problems  in  the  Edmonton  Regional  Office.  The  "Final  Status  Code"  variable  was 
used  as  a  surrogate  to  determine  the  extent  of  the  Control  Sheet  problem.  The  "Final  Status 
Code"  value  is  missing  for  100%  of  the  household  records  from  the  Edmonton  RO.  The  next 
highest  missing  value  rate  is  18%  for  the  Montreal  RO  while  the  remaining  ROs  have 
extremely  low  rates  of  missing  values. 

The  "Received"  variable  captured  as  part  of  the  Control  Sheet  information  is  used  to 
measure  the  mail  response  rate.  A  "l"in  this  field  indicates  that  the  NCT  questionnaire  was 
received  from  the  RO,  meaning  that  the  respondent  had  completed  and  mailed  the 
questionnaire.  It  appears  that  in  the  Edmonton  RO,  only  check  marks  in  the  "Received" 
column  of  the  Control  Sheets  were  recorded  by  interviewers.  When  the  Edmonton  RO 
Control  Sheets  were  data  captured,  the  check  mark  appeared  to  have  been  captured  as  a 
"1".  As  such,  the  missing  values  can  be  assumed  to  represent  cases  in  which  the 
questionnaire  was  not  received  by  mail  in  the  RO.  This  assumption  is  made  for  all  the  mail 


1  Dillman,  D. ,  Sinclair,  M.  and  Clark,  J.  (1993),  "Effects  of  Questionnaire  Length,  Respondent-Friendly 
Design  and  a  Difficult  Question  on  Response  Rates  for  Occupant-Addressed    Census  Mail  Surveys.' 
Public  Opinion  Quarterly  57:  289-3/M. 

2  Unix  files  nct.lfsamp.hhldfile.dat.01.apr21  (LFS)  and  nct.specpop.hhldfile.dat.01.apr.21  (Special). 
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response  rate  estimates  presented  in  Table  2.  That  is,  the  estimates  assumes  that  all  missing 
values  are  cases  in  which  the  questionnaire   was  not  mailed  to  the  RO  by  the  respondent. 

Using  the  Control  Sheet  data,  the  mail  response  rate  for  the  NCT  was  63.5%;  somewhat 
higher  than  the  55.1%  reported  by  SOD.  This  difference  is  accounted  for  by  the  smaller 
denominator  represented  by  the  two  household  files.  While  the  original  intention  was  to 
capture  all  records,  regardless  of  final  status  code,  the  household  files  appear  to  contain 
primarily  the  completed  and  partial  records.  A  frequency  of  "Final  Status  Code"  indicates 
that  70%  of  the  records  are  complete  or  partial  with  the  remaining  29%  missing  values.  The 
Edmonton  RO  accounts  for  approximately  90%  of  these  missing  values.  In  other  words,  the 
records  in  the  household  files  represent  a  sub-set  of  the  denominator  used  by  SOD  to 
calculate  the  mail  response  rate.  It  is  assumed  that  the  "Received"  value  of  "2"  represents 
cases  in  which  the  questionnaire  was  completed  over  the  phone  or  completed/picked-up  by 
the  interviewer  at  the  dwelling. 

If  one  takes  the  SOD  figure  of  9,954  questionnaires  returned  and  uses  the  denominator  of 
15,112  records  data  captured,  the  resultant  is  65.9%.  This  is  fairly  close  to  the  63.5% 
estimate  from  the  Control  Sheet  data.  In  any  event,  the  value  of  Table  2  is  in  a  comparison 
of  the  estimates  among  the  categories.  For  example,  the  mail  response  rate  for  the  LFS 
sample  component  was  15  percentage  points  higher  than  the  mail  response  rate  for  the 
Special  Population  sample.  In  their  study  of  response  rates  in  the  United  States,  Dillman  et. 
al.  (1993)  found  that  low  response  areas  from  the  1990  Census  tended  to  have  lower 
incomes,  lower  levels  of  education,  a  higher  proportion  of  visible  minorities  and  more 
multiple-unit  dwellings  (i.e.  in  urban  areas). 

Table  2 
NCT  Mail,-Back  Response  Rates 


n  =  15,112  [1] 


Mail  Response  Rate 


Total  Sample 


LFS  Component  [3] 


Special  Population  [3] 


"Received"  missing  values 


Contact  at  Drop-Off  [3] 


Yes  Contact 


No  Contact 


"Contact"  missing  values 


Rate  [2] 


63.5% 


66.4% 


50.9% 


3,677 


71.6% 


57.2% 


4,799 
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n  =  15,112  [1] 


Regional  Office  [3] 


St.  John's 


Halifax 


Montreal 


Sturgeon  Falls 


Toronto 


Edmonton 


Vancouver 


"RO"  missing  values 


Rate  [2] 


71.0% 


67.5% 


59.1% 


82.4% 


60.1% 


64.0% 


54.0% 


[1]       The  combined  sample  of  12,273  (LFS)  +  2,839  (Special  Population). 
[2]       Estimate  assumes  all  missing  values  were  not  received  by  the  RO. 
[3]       These  differences  are  statistically  significant  (Chi-Square,  a  =  .01) 

There  is  statistical  evidence  to  support  interviewers'  claims  on  the  debriefing  questionnaire, 
recounted  in  NCT  Report  5  and  NCT  Report  6,  that  contact  with  a  respondent  at  drop-off 
increases  the  likelihood  of  a  respondent  completing  and  returning  the  questionnaire.  Based 
on  the  Control  Sheet  data,  it  would  appear  that  contact  with  the  respondent  resulted  in  a 
mail  response  rate  14  percentage  points  higher  than  for  cases  in  which  contact  was  not  made 
during  drop-off.  This  is  not  unexpected  in  that  mail  survey  research  is  unequivocal  in 
confirming  that  the  most  powerful  influence  on  mail  survey  response  rates  is  the  number  of 
contacts  (Dillman  et.  al-,  1993).  Again,  the  important  implication  of  this  finding  is  for 
Centralized  Edit  procedures  in  which  the  survey  instrument  is  mail-out  /  mail-back. 

The  mail  response  rates  varied  by  Regional  Office  with  the  Vancouver  RO  having  a  rate  10 
percentage  points  below  the  average.  The  highest  mail  response  was  achieved  by  the 
Sturgeon  Falls  RO  with  a  rate  of  over  80%.  It  would  appear  that  two  factors  are 
symbiotically  at  work  in  the  case  of  RO  variation.  The  first  factor  is  that  of  urban  size,  with 
the  Montreal,  Toronto  and  Vancouver  ROs  experiencing  below  average  rates  of  mail 
response.  The  second  factor  is  the  lower  mail  response  rate  in  the  Special  Population 
sample;  all  three  of  these  cities  also  contained  Special  Population  assignments.  It  appears 
that  the  effect  of  each  factor  depends  on  the  area.  For  example,  the  mail  response  rate  in 
the  Halifax  RO  was  25  percentage  points  lower  for  the  Special  Population  sample. 

In  the  Vancouver  RO,  there  was  no  statistically  significant  difference  in  mail  response  rates 
between  the  LFS  and  Special  Populations  samples.  In  the  Montreal  and  Toronto  ROs,  the 
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Special  Population  sample  had  statistically  significant  lower  mail  response  rates  of  5  and  10 
percentage  points  respectively.  In  the  Edmonton  RO,  the  mail  response  rate  was  close  to 
70%  for  the  LFS  sample  but  was  just  over  50%  for  the  Special  Population  sample.  There 
is  also  the  effect  of  RO  training,  procedures  and  personnel  which  is  virtually  impossible  to 
separate  from  the  effects  of  urban  size  and  sample  component. 

Using  the  "Passfail"  variable  captured  as  part  of  the  Control  Sheet  information,  the  edit 
failure  rate  was  virtually  identical  for  the  LFS  and  Special  Population  samples.  However,  the 
edit  failure  rate  followed  a  linear  pattern  from  east  to  west  with  a  low  of  58%  in  the  St. 
John's  RO  to  a  high  of  75%  in  the  Vancouver  RO.  If  the  questionnaire  was  mailed  to  the 
RO,  it  was  statistically  less  likely  to  pass  field  edit.  This  is  not  surprising  given  that  many  of 
the  questionnaires  not  mailed  to  the  RO  would  have  been  completed  by  the  interviewer  over 
the  phone.  As  a  final  note  regarding  the  Control  Sheets,  it  appears  that  many  were  not 
completed  accurately  or  consistently  by  the  interviewers  and,  concomitantly,  this  analysis 
must  be  regarded  with  caution. 

Research  Considerations 

It  is  important  to  recognize  that  these  mail  response  rates  are  lower  than  those  which  would 
be  expected  during  an  actual  Census.  This  is  a  function  of  what  is  referred  to  as  the  "Census 
Climate"  effect;  the  actual  Census  is  supported  by  an  extensive  communications  strategy  to 
raise  public  awareness.  It  is  virtually  impossible  to  quantify  the  effect  of  this  awareness  on 
mail  response  rates.  Perhaps  the  most  important  finding  of  the  Control  Sheet  analysis  is  that 
contact  with  the  respondent  at  drop-off  has  a  significant  impact  on  the  mail  response  rate. 
However,  Centralized  Edit  field  procedures  (mail-out  /  mail-back)  would  eliminate  contact 
at  drop-off.  The  U.S.  Bureau  of  the  Census  has  been  conducting  research  aimed  at 
enhancing  mail  response  rates. 

Dillman  et.  al.  (1993)  report  on  a  test  investigating  several  factors  on  mail  response  rates. 
Using  a  sample  of  17,000  dwellings,  they  tested  5  versions  of  a  census  questionnaire  in  two 
strata.  The  first  stratum  consisted  of  "low  response''  areas  and  the  second  stratum  consisted 
of  "average  response"  areas  based  on  the  1990  Census.  The  results  of  the  test  indicate  that 
a  shorter  questionnaire  resulted  in  a  4%  higher  mail  response  rate  on  average  but  with  a 
much  higher  effect  in  the  "average  response"  stratum.  A  more  respondent-friendly 
questionnaire  was  found  to  have  a  small  effect  in  the  "low  response"  stratum.  However,  the 
interaction  effect  of  these  two  factors  was  estimated  to  result  in  a  mail  response  rate 
approximately  8  percentage  points  higher  in  all  areas. 

The  authors  caution,  however,  that  these  effects  may  be  more  than  offset  by  the  "Census 
Climate"  effect  during  an  actual  Census.  Again,  this  effect  is  impossible  to  quantify.  Finally, 
the  study  tested  the  impact  of  a  difficult  and/or  objectionable  question.  The  study  found  that 
asking  for  Social  Security  Number  resulted  in  not  just  lower  than  average  response  rates  for 
this  particular  question  but  also  lower  mail  response  rates.  In  other  words,  an  objectionable 
question  tends  to  reduce  the  propensity  of  respondents   to  complete  other  questions  and 
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return  the  questionnaire. 

There  are  important  lessons  to  learn  from  this  research.  The  NCT  Special  Population 
sample  is  more  or  less  analogous  to  the  "low  response"  stratum  defined  in  the  U.S.  Census 
Bureau  test  and  households  in  such  areas  were  found  to  be  more  receptive  to  a  respondent- 
friendly  questionnaire.  Clearly  the  combined  effect  of  a  more  friendly  questionnaire  design 
and  fewer  questions  on  mail  response  cannot  be  dismissed,  regardless  of  the  "Census 
Climate".  Finally,  it  should  be  noted  that  the  U.S.  Bureau  of  the  Census  has  also  tested  a 
mandatory  appeal  message  on  the  mail-out  envelopes  and  found  that  this  increased  initial 
response  rates  by  approximately  10%. 3  This  concurs  with  the  feeling  of  several  interviewers 
on  the  NCT  that  if  respondents  were  aware  of  the  mandatory  nature  of  the  test  they  would 
be  more  apt  to  complete  and  return  the  questionnaire. 


4.  Summary 

While  this  report  may  seem  to  consist  of  two  unrelated  sections,  the  results  from  these  two 
sections  may  in  fact  be  "two  sides  of  the  same  coin".  An  analysis  of  the  Interviewer 
Debriefing  Questionnaires  leads  one  to  conclude  that,  in  general,  interviewers  found  the 
NCT  questionnaire  too  long  and  too  difficult  for  respondents.  This  is  quite  apparent  for  the 
Labour  Market  Activities  section.  An  analysis  of  the  NCT  Control  Sheet  data  finds  that 
contact  with  the  respondent  at  drop-off  significantly  increased  mail  response.  However,  in 
preparation  for  a  Centralized  Edit  collection  regime,  Statistics  Canada  should  examine  other 
response-enhancing  strategies.  American  research  indicates  that  mail  response  rates  can  be 
improved  by  designing  a  "friendlier"  and  shorter  questionnaire. 

American  research  also  indicates  that  difficult  and  objectionable  questions  not  only  result 
in  lower  response  rates  for  those  questions  but  also  have  a  discernable,  effect  of  reducing  the 
respondent's  likelihood  of  completing  and  returning  the  questionnaire.  In  the  NCT,  Income 
and  Ancestry  were  found  to  be  both  difficult  and  objectionable  by  many  respondents.  As 
well,  some  respondents  found  the  question  on  household  and  volunteer  activities  to  be 
difficult.  Multiple-part  questions  were  frequently  not  well  reported,  especially  in  the  last 
section  of  the  question.  A  consideration  of  the  issues  examined  in  this  report  seems  to 
indicate  that  the  communications  strategy  will  be  of  paramount  importance  to  the  success 
of  the  1996  Census. 


3  Dillman,  D.,  Clark,  J.,  and  Treat,  J.  (1994),  "Influence  of  13  Design  Factors  on  Completion  Rates  to 
Decennial  Census  Questionnaires",  U.S.Bureau  of  the  Census  1994  Annual  Research  Conference,  Washington. 
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B^trB       Statistics       Statistique 
^m       Canada  Canada 


Canada 


National  Census  Test 

Interviewer  Debriefing  Questionnaire 


December  1993 


Assignment  No. 


Completed  by:     i  O  first-time  STC  Interviewer 

2  O  experienced  STC  Interviewer 


QUESTIONNAIRE  DESIGN 


1.  Overall,  did  the  "Step"  process  work  well? 

3(3  Yes 

4  O  No  -  Please  describe  problems 


2.  Was  the  questionnaire  easy  for  respondents  to  follow? 
bO  Yes 
6  O  No  -  Please  describe  problems 


3.  Were  the  skip  patterns  clear? 

7<0  Yes 

8  O  No  -  Please  describe  problems 


4.  Did  you  often  have  to  rephrase  the  questions  in  simpler  terms  or  discuss  their  meaning  with  respondents? 
1<0  No 
2  O  Yes  -  Please  explain 
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5.  Was  the  layout  of  the  questionnaire  compatible  with  telephone  or  personal  follow-up  interview? 
3<3  Yes 
4  O  No  -  How  did  y°u  resole  this  problem? 


6.  When  completing  the  questionnaires  for  respondents,  did  you: 

s  O  answer  all  the  questions  down  the  page  for  one  person  at  a  time? 

OR 
s  O  answer  the  questions  across  the  page  for  all  persons  at  the  same  time? 


7.  Did  you  get  any  reaction  to  the  message  from  the  Chief  Statistician,  e.g.  confidentiality  of  the  "data  provided  or  the 
mandatory  participation? 

70NO 

8  O  Yes  -  Please  explain 


8.  (a)  Did  most  respondents  fill  in  the  address  box  on  the  front  cover? 
lOYes 

2QNO 


(b)  Did  most  respondents  provide  all  the  information  requested  in  the  address  box? 

3  0Yes 

4  O  No  -  Please  explain 


9.  Do  you  have  any  suggestions  to  improve  the  front  cover? 
5<0No 
6  O  Yes  -  Please  comment 
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COVERAGE-RELATED  QUESTIONS  (Steps  2  to  6) 


10.  In  general,  did  respondents  have  difficulties  distinguishing  between  the  usual  and  the  temporary  residents  of  their  dwelling? 

rONo 

8  O  Yes  —  Please  describe  kinds  of  problems  observed 


11.  Were  there  any  situations  where  it  was  unclear  to  you  or  the  respondent  as  to  whether  a  person  should  be  included  as 
a  usual  resident  of  the  dwelling? 

1ON0 

2  O  Yes  -  Please  give  examples 


12.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

3<ONo 

4  O  Yes  -  Please  indicate  term  and  provide  suggestions  that  would  make  the  term  easier  to  understand,  if  you  have  any 


13.  Did  respondents  correctly  follow  all  the  skip  patterns  of  the  section? 
eO  Yes 
6  O  No  -  Please  describe  problems 


14.  Were  there  any  negative  reactions  from  respondents  to  these  steps? 

7  O  No 

8  O  Yes  —  Please  indicate  which  ones  and  comment 
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FAMILY,  DEMOGRAPHY  AND  ACTIVITY  LIMITATIONS  QUESTIONS  (Questions  2  to  8) 


15.  Were  there  any  questions  in  this  section  that  caused  problems  or  were  difficult  for  respondents  to  answer? 

1  O  No 

2  O  Yes  —  Please  indicate  question  number  and  describe  the  problem 


16.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

3<0No 

4  Q  Yes  -  Please  indicate  question  number  and  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 


17.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 
sONo 
6  O  Yes  ~  Please  indicate  question  number  and  comment 


LANGUAGE  QUESTIONS  (Questions  9  to  11) 


18.  Were  there  any  questions  in  this  section  that  caused  problems  or  were  difficult  for  respondents  to  answer? 

7<0No  <  ■     -: 

8  O  Yes  —  Please  indicate  question  number  and  describe  the  problem 


19.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

1ON0 

2  O  Yes  —  Please  indicate  question  number  and  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 
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20.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 
3ON0 
4  O  Yes  —  Please  indicate  question  number  and  comment 


SOCIO-CULTURAL  INFORMATION  (Questions  12  to  20) 


21.  Were  there  any  questions  in  this  section  that  caused  problems  or  were  difficult  for  respondents  to  answer? 

sONo 

e  O  Yes  -  Please  indicate  question  number  and  describe  the  problem 


22.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

?Ono 

e  O  Yes  —  Please  indicate  question  number  and  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 


23.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 

1<0  No 

2  O  Yes  -  Please  indicate  question  number  and  comment 


24.  Did  respondents  correctly  follow  all  the  skip  patterns  of  the  section? 
aOYes 
4  O  No  -  Please  describe 
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MOBILITY  (Questions  21  and  22) 


25.  Did  respondents  have  any  difficulty  answering  these  two  questions? 

5(3  No 

6  O  Yes  —  Please  indicate  question  number  and  describe  the  problem 


26.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

7(3  No 

8  O  Yes  -  Please  indicate  question  number  and  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 


27.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 

1ON0 

2  O  Yes  —  Please  indicate  question  number  and  comment 


EDUCATION  (Questions  23  to  25) 


28.  Were  there  any  questions  in  this  section  that  caused  problems  or  were  difficult  for  the  respondents  to  answer? 

3ON0 

4  O  Yes  -  Please  indicate  question  number  and  describe  the  problem 


29.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

sONo 

6  O  Yes  -  Please  indicate  question  number  and  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 
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30.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 
7(3  No 
8  O  Yes  -  Please  indicate  question  number  and  comment 


31.  Did  respondents  correctly  follow  all  the  skip  patterns  of  the  section? 
iO  Yes 
2  O  No  —  Please  describe  problems 


HOUSEHOLD  AND  VOLUNTEER  ACTIVITIES  (Question  26  (a),  (b),  (c)  and  (d)  and  Question  27) 


32.  Were  there  any  questions  in  this  section  that  caused  problems  or  were  difficult  for  respondents  to  answer? 

For  example:  Were  the  response  categories  adequate?  Were  the  examples  helpful?  Did  respondents  have  any  trouble  deciding 
In  which  part  they  should  Include  an  activity? 


3ON0 

4  O  Yes  ~  Please  indicate  question  number  and  part  and  describe  the  problem 


33.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

For  example:  seniors,  children,  housework,  non-profit  organization. 

sONo 

e  O  Yes  —  Please  indicate  the  question  number  and  the  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 


34.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 
7<ONo 
8  O  Yes  —  Please  indicate  question  number  and  comment 
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LABOUR  MARKET  ACTIVITIES  (Questions  28  to  45) 


35.  Were  there  any  questions  in  this  section  that  caused  problems  or  were  difficult  for  respondents  to  answer? 

1  O  No 

2  O  Yes  —  Please  indicate  question  number  and  describe  the  problem 


36.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 
For  example:  manufacturing,  wholesale  and  retail  trade. 

3  O  No  ~ 

4  O  Yes  -  Please  indicate  the  question  number  and  the  term  and  provide  suggestions  that  would  make  the  term  easier  to 

understand,  if  you  have  any 


37.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 

sONo 

6  O  Yes  —  Please  indicate  question  number  and  comment 


38.  Did  respondents  correctly  follow  all  the  skip  patterns  in  this  section? 
?OYe3 
8  O  No  -  Please  indicate  question  number  and  describe  problems 


Page  8 


39.  Did  respondents  indicate  any  resentment  or  confusion  in  answering  Question  36  in  those  cases  where  they  were  required 
to  indicate  "Other",  rather  than  a  specific  industry  category? 

1ON0 

2  O  Yes  -  Please  describe  the  problem 


40. 


Did  respondents  indicate  any  difficulty  in  determining  if  their  activities  should  be  reported  in  the  HOUSEHOLD  AND  VOLUNTEER 
ACTIVITIES  section  (Questions  26  and  27)  or  in  the  LABOUR  MARKET  ACTIVITIES  section  (Questions  28  to  45)? 

For  example,  In  Question  39  was  there  any  confusion  noted  In  the  second  category,  working  without  pay  for  his/her 
spouse  or  another  relative  In  a  family  farm  or  business. 

3(0  No 

4  O  Yes  -  Please  describe  difficulty  or  confusion 


INCOME  (Question  46) 


41.  Were  there  any  parts  in  this  question  that  were  difficult  for  respondents  to  answer? 
sONo 
6  O  Yes  -  Please  indicate  part  and  describe  problem  in  as  much  detail  as  possible 


42.  Were  there  any  negative  reactions  from  respondents  to  this  question? 
7(0  No 
8  O  Yes  -  Please  describe  and  indicate  the  frequency,  along  with  your  comments 
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DWELLING  (Questions  47  and  48) 


43.  Were  there  any  questions  in  this  section  that  were  difficult  for  respondents  to  answer? 

1  O  No 

2  O  Yes  -  Please  indicate  question  number  and  describe  the  problem 


44.  Were  there  any  terms  in  this  section  that  were  not  well  understood  by  respondents? 

3<ONo 

4  O  Yes  -  Please  indicate  question  number  and  term  and  provide  suggestions  that  would  make  the  term  easier  to 
understand,  if  you  have  any 


45.  Were  there  any  negative  reactions  from  respondents  to  these  questions? 

sONo 

6  O  Yes  -  Please  indicate  the  question  number  and  comment 


FIELD  OPERATIONS 


46.  Did  some  respondents  want  to  complete  the  questionnaire  during  Drop-off? 

7  O  No 

8  O  Yes  -  How  many?  ► 


47.  Were  the  pictures  for  the  types  of  dwelling  codes  on  your  reference  card  any  help  in  deciding  which  code  to  use? 

1  O  Yes 

2  O  No  -  Indicate  the  least  helpful  and  explain 


48.  Were  the  definitions  for  each  type  of  dwelling: 

3  O  easily  understood 

4  O  confusing  -  Indicate  which  ones  and  explain 
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49.  Did  you  use  the  Dwelling  Type  Chart  on  your  reference  card? 

s  O  Yes  -  How  often? 

e  O  A  few  times 
7  O  Often 

8  O  No  -  Why  not? 


50.  Did  you  refer  to  the  Interviewer's  Manual  during  enumeration? 
1  O  Yes  ~  Indicate  the  reason  and  explain 


2  O  No  -  Why  not? 


51.  Was  it  easily  understood? 

3  O  Yes  —  easily  understood 

4  O  No  -  confusing 

If  no,  indicate  which  parts  and  explain  confusion. 


52.  Could  you  find  answers  quickly  in  the  manual? 
s  O  Yes 
6(0  No 


53.  Do  you  have  any  suggestions  that  would  make  the  manual  a  better  reference  tool  and  easier  to  use? 
7(0  No 
a  O  Yes  -  Please  comment 
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54.  Were  any  of  the  procedural  definitions  confusing? 

1  O  No 

2  O  Yes  -  Indicate  which  ones  and  explain 


55.  Were  the  edit  steps  easy  to  understand? 

3(3  Yes 

4  O  No  -  Indicate  which  ones  and  explain 


56.  Did  you  complete  the  edit  for  each  person  first  or  for  each  question  before  doing  the  next  one? 
5  O  Person 
s  O  Question 


57.  When  did  you  refer  to  the  edit  steps  while  editing  questionnaires? 

1  O  At  the  beginning 

2  O  Most  of  tne  time 

3  O  F°r  specific  questions 

4  O  All  the  time 


Explain 


58.  Were  the  rules  for  determining  if  Follow-up  is  required  clear? 
5  O  Yes 
e  O  N°  -  Indicate  why  not  and  explain 
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59.  Would  any  other  tool(s)  have  helped  you  do  your  job  better? 
7ON0 
8  O  Yes  _  Indicate  what  they  are  and  explain 


OTHER  COMMENTS 


60.  Write  any  further  comments  or  suggestions  that  you  may  have  to  improve  the  NCT  questionnaire. 


Thank  you  very  much. 

Your  comments  and  suggestions  will  be  carefully  considered  for  the  1996  Census. 
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APPENDIX  B 


Debriefing  Questionnaire  Responses 


N  =  263 

Yes  (%) 

No(%) 

No  Response 

Questionnaire  Design 

Question  1 

79.2% 

20.8% 

8 

Question  2 

68.3% 

31.7% 

20 

Question  3 

52.2% 

47.8% 

12 

Question  4 

48.4% 

51.6% 

13 

Question  5 

85.5% 

14.2% 

23 

Front  Page 

Question  7 

28.5% 

71.5% 

10 

Question  8a 

98.1% 

1.9% 

4 

Question  8b 

94.2% 

5.8% 

4 

Question  9 

18.3% 

81.7% 

11 

Coverage-Related    (Steps  2  to  6) 

Question  10 

12.6% 

87.4% 

9 

Question  11 

22.9% 

77.1% 

10 

Question  12 

14.7% 

85.3% 

12 

Question  13 

71.6% 

28.4% 

13 

Question  14 

12.2% 

87.8% 

8 

Family,  Demography  and  Activity 
Limitations  (Questions  2  to  8) 

Question  15 

41.8% 

58.2% 

12 

Question  16 

20.0% 

80.0% 

13 

Question  17 

25.9% 

74.1% 

8 
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N  =  263 

Yes(%) 

No  (%) 

No  Response 

Language  (Questions  9  to  11) 

Question  18 

7.8% 

92.2% 

5 

Question  19 

4.7% 

95.3% 

6 

Question  20 

5.5% 

94.5% 

8 

Socio-Cultural  (Questions  12  to  20) 

Question  21 

44.2% 

55.8% 

12 

Question  22 

21.7% 

78.3% 

9 

Question  23 

32.8% 

67.2% 

10 

Question  24 

82.1% 

17.9% 

12 

Mobility  (Questions  21  and  22) 

Question  25 

6.2% 

93.8% 

5 

1  Question  26 

7.0% 

93.0% 

5 

Question  27 

3.9% 

96.1% 

5 

Education  (Questions  23  to  25) 

Question  28 

13.6% 

86.4% 

6 

Question  29 

8.6% 

91.4% 

8 

Question  30 

5.5% 

94.5% 

9 

Question  31 

77.4% 

22.6% 

11 

Household  &  Volunteer  Activities 
(Questions  26  and  27) 

Question  32 

36.4% 

63.6% 

10 

Question  33 

11.4% 

88.6% 

9 

Question  34 

30.7% 

69.3% 

9 

Labour  Market  Activities  (Questions  28  to  45) 

Question  35 

57.2% 

42.8% 

20 

Question  36 

23.9% 

76.1% 

12 

Question  37 

31.6% 

68.4% 

13 

-34- 


N  =  263 

Yes(%) 

No(%) 

No  Response 

Question  38 

46.4% 

53.6% 

11 

Question  39 

23.4% 

76.6% 

7 

Question  40 

9.5% 

90.5% 

10 

Income  (Question  46) 

Question  41 

63.6% 

36.4% 

10 

Question  42 

95.3% 

4.7% 

9 

Dwelling  (Questions  47  and  48) 

Question  43 

5.5% 

94.5% 

8 

Question  44 

3.1% 

96.9% 

8 

Question  45 

7.5% 

92.5% 

9 

Field  Operations 

Question  46 

42.5% 

57.5% 

11 

Question  47 

91.2% 

8.8% 

14 

Question  49 

84.3% 

15.7% 

27 

Question  50 

72.9% 

27.1% 

16 

Question  51 

86.6% 

13.4% 

24 

Question  52 

76.8% 

23.2% 

17 

Question  53 

37.7% 

62.3% 

43 

Question  54 

12.6% 

87.4% 

25 

Question  55 

84.0% 

16.0% 

20 

Question  58 

84.6% 

15.4% 

16 

Question  59 

42.0% 

58.0% 

37 

Continued 
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Appendix  B  (continued) 


Question 


Question  48 


Question  49 


Question  56 


Question  57 


Response  Category 


Question  06  down  the  page  for  one  person, 

across  the  page  for  all  persons, 
missing 


Question  46  How  many  (average) 


easily  understood 

confusing 

missing 


A  few  times 
Often 


Person 

Question 

missing 


At  the  beginning 
Most  of  the  time 
For  specific  purposes 
All  the  time 
missing 


Question  60  other  comments 


50.6% 

49.4% 

28 


5.03 


83.9% 

16.1% 

9 


63.3% 
36.7% 


56.5% 

43.5% 

17 


50.2% 

14.3% 

27.3% 

8.2% 

18 


202 
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APPENDIX  C 
NCT  Reports 


1.  Proposal  for  the  1996  National  Census  Test 

2.  1996  Census  Consultation  Report  /  Rapport  sur  les  consultations 
du  recensement  de  1996 

3.  Questionnaire   Design  Consultation 

4.  Evaluation  of  the  Census  Help  Line 

5.  Interviewer  Debriefing  Report 

6.  Evaluation  of  Field  Collection  Procedures 

7.  Automated  Coding  Report 

8.  Rapport  sur  le  niveau  de  langue  des  questionnaires 

9.  Evaluation  of  Questionnaire   Design  and  Layout  (CRC) 

10.  Edit  Failure  and  Response  Rates 

11.  Coverage-Related   Steps  and  Step  9 

12.  Agriculture  Operator 

13a.  Evaluation  des  variables  demograpbiques 

13b.  Family  Characteristics 

14.  Analyse  des  donnees  linguistiques 

15.  Place  of  Birth,  Citizenship  and  Immigration 

16.  Ethnic  Origin 

17.  Aboriginal  Report 

18.  Population  Group 

19.  Mobility  and  Migration 

20.  Household  and  Volunteer  Activities 

21.  Labour  Market  Activities 

22.  Place  of  Work  /  Mode  of  Transportation 

23.  Analysis  of  Income  Components 

24.  Household  Maintainer 

25.  Evaluation  of  Respondents'   Comments 

26.  Evaluation  of  Questionnaire   Design  and  Layout  (STC) 

27.  Interviewer  Debriefing  Questionnaires   and  Mail  Response  Rates 

28.  Evaluation  of  Processing  Operations 
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